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Abstract 


N o menclat ure 


Two methods of creating approximation models are 
compared through the calculation of the modeling 
accuracy on test problems involving one, five, and 
ten independent variables. Here, the test problems 
are representative of the modeling challenges typi- 
cally encountered in realistic engineering optimiza- 
tion problems. The first approximation model is a 
quadratic polynomial created using the method of 
least squares. This type of polynomial model has 
seen considerable use in recent engineering optimiza- 
tion studies due to its computational simplicity and 
ease of use. However, quadratic polynomial models 
may be of limited accuracy when the response data 
to be modeled have multiple local extrema. The sec- 
ond approximation model employs an interpolation 
scheme known as kriging developed in the fields of 
spatial statistics and geostatistics. This class of inter- 
polating model has the flexibility to model response 
data with multiple local extrema. However, this flex- 
ibility is obtained at an increase in computational 
expense and a decrease in ease of use. The intent 
of this study is to provide an initial exploration of 
the accuracy and modeling capabilities of these two 
approximation methods. 

Keywords: approximation, response surface model, 
polynomial model, kriging, DACE 


ANOVA 

analysis of variance 

c 

vector of unknown coefficients in least 
squares surface fitting 

c 

vector of estimated coefficients in least 
squares surface fitting 

DACE 

design and analysis of computer 
experiments 

/(■> 

unknown function 

/(') 

predicted function 

f 

vector of constants used in DACE 
models 

MSE 

mean squared error 

n c 

number of candidate sample sites 

n e 

number of sample sites to calculate 
modeling error 

n s 

number of sample sites in design space 

n t 

number of terms in a polynomial model 

n v 

number of design variables 

r 

vector of correlation values 

R(-) 

correlation function 

R 

correlation matrix 

RMS 

root mean square error 

RMS u b 

unbiased root mean square error 

RS 

response surface 

RSM 

response surface methodology 

X 

scalar component of x 

X 

vector denoting all locations in 
n^-dimensional space 

x (p) 

vector denoting the p th location in 
rt„-dimensional space 
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x ip ) vector of the polynomial model terms 

at the p th sample site 

X matrix of sample site locations in least 

squares surface fitting 
y scalar observed response value 

observed response value at the p th 
sample site 

y mean of observed response values 

y scalar predicted response value 

?/(•) prediction function 

y vector of observed response values at 

sample sites 

y vector of predicted response values 

Z(-) Gaussian random function 

/? parameter in DACE modeling 

/ 3 estimated DACE modeling parameter 

5 mean modeling error 

6 max maximum modeling error 

< 5 median median modeling error 
e parameter used in defining the DACE 

test function 

Tip prior distribution on (3 

erg standard deviation of modeling error 

a 2 sample variance 

a 2 estimated sample variance 

9 scalar correlation parameter used in 

DACE modeling 

1 Introduction 

Approximation models are often used in engineering 

rogates, has grown in popularity, a variety of model 
ing methods have been employed. Perhaps the most 
popular techniques involve polynomial models, typ- 
ically linear or quadratic functions, created by per- 
forming a least squares curve fit to a set of data, 
where the data consist of one or more dependent re- 
sponse values along with one or more independent 
variables. Collectively, these polynomial-based mod- 
eling methods have come to be known as response 
surface models which is a term taken from the sta- 
tistical literature (cf., Myers and Montgomery [1, 
pages 1-10]). These methods are popular for a num- 
ber of reasons one of which is that they provide a 
compact and explicit functional relationship between 
the response and the independent variables. In ad- 
dition, the method of least squares used in creating 
the models is relatively computationally inexpensive 
and straightforward. Evidence of the popularity of 
polynomial modeling is given by the wealth of recent 
reports both by the authors of this document [2] and 


by others as well (cf., [3, 4, 5, 6, 7, 8, 9]). 

Originally these polynomial modeling methods 
were developed to produce smooth approximation 
models of response data contaminated with random 
error found in typical physical (stochastic) experi- 
ments. Due to the ease of use of the polynomial mod- 
eling methods, these techniques migrated to the field 
of deterministic computer experiments where there 
is no random error (i.e., response data are identical 
each time the simulation is repeated). The applicabil- 
ity of using these methods in modeling deterministic 
response data is the subject of debate in the statisti- 
cal community, some aspects of which are addressed 
by Simpson, et al [10]. 

In response to this issue, Sacks, et al [11] pro- 
posed the use of interpolation models to approxi- 
mate response data obtained from deterministic com- 
puter simulations. Their interpolation models are 
based on techniques known as kriging originally de- 
veloped in the fields of spatial statistics and geostatis- 
tics as described by Cressie in [12, pages 1-26] and 
[13]. (The term kriging was first used in the work of 
Matherton [14] who attributed the original develop- 
ment of the statistical techniques to mining engineer 
D. G. Krige.) 

Recent studies by researchers at Boeing includ- 
ing Frank [15] and Booker [16] have employed DACE 
modeling methods in engineering optimization prob- 


trends whereas quadratic polynomials are by defini 
tion unimodal. However, the flexibility offered by 
DACE modeling methods is offset by the lack of an 
explicit model function as well as an increase in com- 
putational expense over that incurred in polynomial 
modeling. 

The purpose of the research described here is to 
compare the modeling accuracy of both polynomial 
models and DACE models on a set of sample test 
problems. To the authors’ knowledge, such a com- 
parison has not been reported elsewhere and this in- 
vestigation will provide useful quantitative and qual- 
itative data on the utility of these modeling methods. 
In Section 2 the mathematical underpinnings of poly- 
nomial and DACE modeling are presented. Section 3 
contains the description of the test problems and Sec- 
tion 4 contains an assessment of modeling accuracy 
for the test problems. A summary of this research is 
presented in Section 5. 
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2 Approximation Model For- 
mulation 

2.1 Background on Approximation 
Models 

Prior to a description of the mathematical underpin- 
nings of the approximation modeling methods, it is 
useful to compare the philosophy of polynomial mod- 
eling methods to that of DACE interpolating meth- 


models based on Bayesian statistics and kriging. Al- 
though both RS models and DACE models are ap- 
proximations to the true, unknown response surface 
and as such are technically response surface mod- 
els, the statistical literature tends to reserve the term 
response surface model for polynomial models. The 
phrase polynomial RS model will be used to reinforce 
this distinction. 

The construction of polynomial RS models or 
DACE interpolating models relies on the sampling 
of the design space at n s unique locations in the de- 
sign space to obtain response values for the objective 
function or the constraints. Here, the design space is 
defined by the upper and lower bounds on the vector 
x of n v independent variables, where 

x = [xi,x 2 ,...,x nv \. (1) 

The upper and lower bounds create a design space in 
the shape of an n^-dimensional cube which has 2 n ” 
vertices. Note that experimental error is not present 
when obtaining results from deterministic computer 
models. Thus, no information is gained from the re- 
peated sampling of the same location in the design 
space. From the sampled data approximation mod- 
els are constructed to describe the variation in the 
response (s) with respect to the n v independent vari- 
ables. Mathematically, the true underlying functional 
relationship is expressed as 

y = /( x )i ( 2 ) 

where y is the observed response and /(x) is the un- 
known function. 

In many engineering optimization problems the 
cost of computing the objective function or con- 
straints is computationally expensive. For this rea- 
son, approximation models are employed in the op- 
timization problem as surrogates for these expensive 
function evaluations. These approximation models 
are expressed as 

2 /=/(x). (3) 


Polynomial RS models can be thought of as 
“global” models in which all of the n s observed values 
of the response are equally weighted in the fitting of 
the polynomial surface. At an unsampled location in 
design space, x, response observations that are near 
to x (in the sense of Euclidean distance) have an equal 
influence on the predicted response, /(x), as do the 
response observations that are far from x. It may be 
argued that such a global model may not be the best 
approximator if the true unknown response has mul- 
illli hi i i 


less strongly influenced by those further away. Such 
local modeling behavior is characteristic of interpola- 
tion models, of which DACE models are one partic- 
ular implementation. 

2.2 Polynomial Approximation Mod- 
els 

Polynomial response surface modeling (RSM) em- 
ploys the statistical techniques of regression analysis 
and analysis of variance (ANOVA) to determine /(x) 
through a systematic decomposition of the variabil- 
ity in the observed response values. The empirical 
model is then estimated by assigning portions of the 
variability to either the effect of an independent vari- 
able or to random error. 

If n s analyses are conducted and p = 1 ,...,n s , 
then a quadratic response surface (RS) model has the 
form 

y (p) = °o + ^2 c j x< j p) + 

1<j <nt, 

^ ] c (n v -l+j+k) x j x k > (4) 

1 <i<fc<n„ 

where y ^ is the response; x ^ and x'^ ! are the n v 
design variables; and c 0 , Cj, and C( nu _i + j + /-) are the 
unknown polynomial coefficients. Note that there 
are n t = (n v + l)(n„ + 2) / 2 modeling terms in the 
quadratic polynomial. This polynomial model is writ- 
ten in matrix notation as 

y (p) =c T x(p) ! (5) 

where c is the vector of length n t of unknown coeffi- 
cients to be estimated, 

c — [cq , C\ , . . . , c n( _ 2 ] , ( 6 ) 


3 

American Institute of Aeronautics and Astronautics 



3 k 

model (Equation 4). For the p th observation this is 

5 c(p) — ri r (p) r (p) ■■■ r (p) r (p) ••• (A p '>) 2 1 (71 

A , ^2 5 5^1 '*'2 5 )rnj J- Vv 

h h i diff b h th 


to the probability density function which one assigns 
to a variable of unknown value before any experimen- 
tal data on that variable are collected [20, pages 4,5]. 
The prior distribution is the mechanism in Bayesian 


Estimating the unknown coefficients requires n s 
analyses, where n s > n*. Under such conditions, the 
estimation problem is formulated in matrix notation 
as 

y « xe (8) 


and X is the matrix formed by the p row vectors 
x* 1 ) , . . . , x (p ) which is assumed to have rank n t . Thus, 
X is expressed as 


l J 

The unique least squares solution to Equation 8 
is 

c = (X T X)- 1 X T y, (11) 


This intentional bias is the source of much contro- 
versy in the statistical community. In spite of the 
differences between classical statistics and Bayesian 
statistics, Berger [20, pages 109,110] emphasizes that 
both classical and Bayesian statistics have merit and 


In the DACE literature the true, unknown func- 
tion to be modeled is typically expressed as 

«/(x) = /(x) + Z(x), (13) 


) ( ) 

Equation 13 in some sense is a “global” model for 
the entire design space based on the n s response ob- 
servations, while the Z(x) term creates a “localized” 
deviation from the global model. 


this corresponds to 

y = c T x. (12) 

Note that if n s > % the system of equations is overde- 
termined. Thus, the pi'edicted response values (from 
the polynomial RS model) at the original sample lo- 
cations may differ from the observed response values 
at the sampled locations. 

2.3 DACE Approximation Models 

The objective here is to provide an introduction to 
the statistics and mathematics of DACE modeling. A 
detailed treatment of the statistical and mathemati- 
cal methods involved in DACE modeling is found in 
the work of Sacks et al. [11]; Koehler and Owen [17]; 
Osio and Arnon [18]; and Booker et al. [19]. 

Before addressing the principles underlying 
DACE modeling methods, it is useful at this point to 


y(x) = fd + Z{x). (14) 

The term f3 takes on different meanings depending 
on one’s statistical point of view. From the perspec- 
tive of the kriging approach used in DACE, f3 is an 
unknown constant to be estimated based on the n s 
observed response values. From the perspective of 
Bayesian statistics (3 is a random variable with a prior 
distribution denoted as np. The interpretations of (3 
are identical regardless of the statistical perspective 
if Z(x ) has a Gaussian distribution and irp has a uni- 
form distribution [11], i.e., if no prior knowledge is 
used to provide an initial estimate for (3. 

The covariance matrix of Z(x) is expressed as 

Cov[Z(x^), Z(x (j '))] = <t 2 R[R( X W, x^)], (15) 

where R is the correlation matrix , and R is the cor- 
relation function which is selected by the user. In 
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Equation 15 i = 1 ,...,n s and j = 1 ,...,n s . Note 
that the correlation matrix R is symmetric with val- 
ues of unity along the diagonal. 

As noted above, the user may select the form of 
the correlation function R. Sacks et al. [11]; Koehler 
and Owen [17]; and Booker et al. [19] provide a de- 
tailed description of various correlation functions that 
may be used. A choice for R often found in the sta- 
tistical literature, and employed in [19], is an expo- 
nential correlation function 

n v 

R(x {, \x b) ) = exp[-^6>fc|xj. !) -x[ j) | 2 ], (16) 

k = 1 

where Ok is the vector of unknown correlation pa- 
rameters. For this research only a single correlation 
parameter is used instead of a vector of correlation 
parameters. The scalar correlation parameter is de- 
noted as 0. Thus, Equation 16 may be rewritten as 

fl(x W ,x (,,) ) = ex PMElx^-x^l 2 ], (17) 

k = l 

The process by which a value for 0 is estimated is 
given below. 

Another term of interest is the correlation vector, 
r(x), between the response at a location, x, and the 
xi 1 ) , . . . , x* n d response values. The correlation vec- 
tor is expressed as 

r(x) = 77(x,x«) 

= [R(x, x^), R(x, xi 2) ), • • • , R(x, x( n '>)]. 

(18) 

While Equation 14 represents the true, unknown 
function to be approximated, the computed (i.e. , es- 
timated) DACE model is given the symbol y(x). In 
statistical notation, this estimated DACE model is 
defined as 

y(x) = E(y(x)[y(x (1) ), . . . , y(x l ' ns '>)) i (19) 

where the expression E(-) is the statistical symbol for 
the expected value of (•) and the expression E(A\B) is 
the expected value of A given the information B. The 
terms y(x t ' 1 ' 1 ), . . . , y(x( n ^) are the n s observed values 
of the response, y(x.) is the true response one is at- 
tempting to estimate, and y(x) is the actual estimate 
of the response (which one hopes is close to y(x)). 
This distinction between y(x) and y(x) is necessary 
so that the concept of mean squared error (MSE) may 
be introduced where 

MSE = E(y(x)-y(x)) 2 . (20) 


This is simply a measure of the amount of error be- 
tween the DACE model, y(x), and the true model, 
t/(x), at all locations, x, in the design space. Since 
the DACE model performs interpolation there is no 
error between the DACE model and the true model 
at the n s sites where the values of the response are 
known. 

If MSE is minimized, y(x) becomes 

y(x) = /? + r T (x)R -1 (y — pi), (21) 

where (3 is unknown, and both r(x) and R depend 
on the unknown parameter 0. Note that the vector f 
has length n s with all entries equal to unity 

f=[l, •••,!], (22) 

which is a result of the assumption that all of the 
variability in y(x) is accounted for in the Z(x ) term. 
While the usual notation for a vector with all entries 
equal to unity is e, the vector f is retained to main- 
tain similarity with the notation used in Koehler and 
Owen [17] and Booker et al. [19]. 

The unknown parameter 9 is found using maxi- 
mum likelihood estimation as described by Booker et 
al. [19]. In this approach, the values for (3 and the es- 
timated variance, a 2 , are obtained using generalized 
least squares as 

/?= (f r R- 1 f)- 1 f r R- 1 y, (23) 

and 

&2= (y-Pf) T R-\y-ft). 

n s 

Note that Equations 23 and 24 implicitly depend on 
the correlation parameter 0. 

The maximum likelihood estimation of 9 is re- 
duced to a one-dimensional optimization problem 
with simple bounds of the form 

max (— 1/2) [(n s In cr 2 ) + In |R|] , 
seR 1 

subject to 0 < 9 < oo. (25) 

Thus, by solving this one-dimensional optimization 
problem the DACE approximation model y(x) is 
completely defined. Note that if Equation 16 were 
used (i.e., retaining a vector of correlation parame- 
ters), then the one-dimensional minimization prob- 
lem becomes an n v —dimensional minimization prob- 
lem. 
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3 Approximation Model Test 
Problems 

The objective of performing the test problems was 
to gain an understanding of the strengths and weak- 
nesses of DACE modeling as compared to polynomial 
RS modeling. For these efforts two test problems 
were formulated where the first test problem was ex- 
pected to be biased in favor of the DACE modeling 
method and the second test problem was expected 
to be biased in favor of the polynomial RS model- 
ing method. A critical element of this comparison is 
the investigation of how the accuracy of the DACE 
models and RS models is affected as the number of 
dimensions, n v , increases. To investigate this aspect 
of modeling accuracy, test problems involving n v = 1, 
n v = 5, and n v = 10 were examined. 


3.1 Test Problem Formulation 

For this investigation a simple test function was cho- 
sen so that it could be exhaustively examined with 
minimal computational expense. This test function 
has the form 


3 . ,16 . . 2 A6 . 

— + s m (— aj* — e) + sm (—Zi-e) , 

(26) 

where the term e acts as a shifting mechanism 
to make the response, t/(x), appear more or less 
quadratic on the range [—1, l] n, \ The values used for e 
are described below. Since there is no numerical noise 
inherent in Equation 26 it is henceforth referred to as 
the smooth test function. 

To simulate the effects of numerical noise often en- 
countered in realistic engineering optimization prob- 
lems (cf., [2], [21], [22], [23], [24]), a high-frequency 
low-amplitude sine wave function was added to Equa- 
tion 26. This noisy test function has the form 


y( x ) = £ 


2/(x) 


£ 

i = 1 L 


3 .16 . . 2 ,16 

+ sm (j-^ - <0 + sin (— Xi 


100 


16 


— sin (40(— ^-e) 


t)+ 

(27) 


where the term on the far right of Equation 27 is the 
high frequency, low amplitude component. 

The first test function (Case 1) was created for 
e = 1.0 and a plot of the noisy version of this func- 
tion for n v = 1 is shown in Figure 1. Both the smooth 
and noisy variants of the Case 1 test functions are 
shown in Figure 2 for n v = 2. This function appears 
quasi-sinusoidal on [— 1 , 1] . 


The Case 2 test function was created using e = 0.7 
and has a quasi-quadratic trend on [— 1 , 1] . The noisy 
Case 2 test function is shown in Figure 3 for n v = 1 
and both the smooth and noisy Case 2 test functions 
are shown in figure 4 for n v = 2. 


3.2 Evaluation of Modeling Accuracy 

For both Cases 1 and 2, DACE and RS models (de- 
noted as yfx)) were constructed based on n s evalu- 
ations (response values) of the noisy test function. 
These models were then used to estimate the un- 
known response values of the smooth test function at 
n e locations, where n e 3> n s . These predicted smooth 
function response values are denoted as y„ e . To eval- 
uate the accuracy of the DACE and RS models, the 
actual response values of the smooth test function 
are also calculated for the n e locations. These actual 
smooth function response values are denoted as f/„ e . 
The discrepancy between y„ e and y„ e is known as the 
modeling error. Note that the definition of the mod- 
eling error is different from the residual error which is 
the discrepancy between a polynomial model and the 
data points in an overdetermined least squares prob- 
lem. There is no residual error in DACE modeling 
since the DACE method exactly interpolates the n s 
response values. 

The total modeling error in the DACE and 
RS models is characterized using five error met- 
rics. These are the mean error, <5, the median error, 
<5 median > the standard deviation, cr^ , the maximum 
error, S max , and the unbiased RMS error RMS u b- 

In these error metrics the modeling error is de- 
fined as 

Si = \m-yi\, (28) 

for i = 1, . . . , n e . Using this notation, the mean mod- 
eling error is 

S = -TS t , (29) 

and the standard deviation of the modeling error is 


C5 


n P — 1 


(30) 


The median modeling error, S me dian , is defined as the 
midpoint value of the series in which the values 
are placed in ascending value. The maximum value 
of this series is the maximum modeling error, 8 max , 
which is defined as 

8max — max(A). (31) 
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In addition to these metrics, the root mean squared 
modeling error is 



If the n e locations are not the same as the sample 
sites, n s , used to construct the approximation model, 
then Equation 32 is an unbiased estimator of the RMS 
modeling error and is identified as RMS u b- However, 
if n e and n s are the same, then Equation 32 is bi- 
ased, and it underestimates the true error. When n e 
and n s are the same the unbiased RMS error must be 
calculated using 


RMS ub 



(33) 


accurate as the mean-value model. In addition, the 
DACE interpolation model has the property that it 
becomes a mean-value model when it is used to in- 
terpolate far from any sample sites. Note that in the 
one variable test problem the mean-value models are 
y = 0.2342 for Case 1 and y = 0.2680 for Case 2. 

The modeling errors for DACE, polynomial RS, 
and mean-values models were calculated using the 
201 values of the smooth test function. The results 
for Cases 1 and 2 are shown in Table 1 . For Case 1 the 
DACE model is more accurate than both the polyno- 
mial RS model and the mean-value model, whereas 
for Case 2 the polynomial RS model is more accurate 
than the other two models. This corresponds to the 
trends shown in Figures 5 and 6. 

4.2 Five Variable Test Problem 


where nt is the number of terms in the polynomial 
model. See Myers and Montgomery [1, page 26] for 
more information on unbiased estimators. 

4 Results 

4.1 One Variable Test Problem 

In the one variable test problem the test functions 
were sampled at three locations ( n s = 3) -0.5, -0.3, 
and 0.7 on [—1, 1], From the response values at these 
three sites a DACE model and a quadratic polyno- 
mial RS model were created. To test the accuracy 
of the DACE and RS models, the smooth test func- 
tion was sampled at 201 equally spaced points along 
[— 1, 1]. Figures 5 and 6 show the DACE and RS mod- 
els used in Cases 1 and 2, respectively. 

For the Case 1 test problem the DACE model had 
a correlation parameter of 9 = 7.540 and (3 = 0.2127, 
while for the Case 2 test problem these values were 
9 = 28.394 and /? = 0.2777. The quadratic response 
surface polynomial models for both the Case 1 and 
Case 2 test functions were created using the Fit [■] 
function in Mathematica [25, pages 859-861]. 

In addition to a DACE model and a quadratic 
polynomial RS model a third model was examined 
where 

y(x) = y, (34) 

where y is the mean of the n s observed response val- 
ues in y. This mean- value model was selected since it 
represents what is perhaps the most simple, computa- 
tionally inexpensive, approximation model one may 
create. Further, it provides a sort of lower bound 
on modeling accuracy, i.e., one would expect a more 
“complex” approximation model would be at least as 


For the five variable test problem n s = 50 and 
n e = 3125. The 50 sample sites correspond to those 
obtained from a D-optimal experimental design used 
in previous research related to this work (see Giunta, 
et, al [2]), with all of the sample sites contained in 
the domain defined by [ — 1 , 1] 5 . The 3125 test sites 
were created by discretizing the design space into a 
5 x 5 x 5 x 5 x 5 mesh where 5 5 = 3125. 

For the Case 1 test problem the DACE model had 
a correlation parameter of 9 = 0.45 and /3 = 1.3516, 
while for the Case 2 test problem these values were 
9 = 0.08 and j3 = 5.9593. As above, the quadratic re- 
sponse surface polynomial models for the Case 1 and 
Case 2 test functions were created using the Fit[-] 
function in Mathematica. 

In addition to a DACE model and a quadratic 
polynomial RS model, two other approximation 
methods were examined. The third model is a com- 
bined RS/DACE model of the form 

y(x) = /(x) -(- Presiduai + Z(residual), (35) 

where /(x) is the quadratic polynomial RS model 
found using Mathematica and ftresidual+Z (residual) 
is a DACE model applied to the residual error exist- 
ing in the least squares surface fit for /(x). For the 
Case 1 RS/DACE model the optimal correlation pa- 
rameter was 9 = 30.0, and /? = —5.25 • 10“ '. In the 
RS/DACE model for Case 2 these parameters were 
9 = 30.0, and (3 = —5.76 • 10“ ' . The fourth model ex- 
amined is the mean- value model (Equation 34) where 
for Case 1, y = 1.2512 and for Case 2, y = 2.1418. 

The modeling errors for these four approximation 
models were calculated for Case 1 and Case 2 test 
functions and are listed in Table 2. In the Case 1 
results the polynomial RS model and the combined 
RS/DACE model have nearly identical values for the 
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modeling errors. For the DACE method the modeling 
error is not as low as for the polynomial-based mod- 
els, but it is lower than for the mean-value model. 
Similar trends are exhibited in the Case 2 results 
where the modeling errors for the polynomial R,S 
model and the RS/DACE model are nearly identi- 
cal, are the modeling errors for the DACE model are 
only marginally worse. In Case 2 however, the mean- 
value model has considerable higher modeling errors 
than the other three models. 


4.3 Ten Variable Test Problem 

For the ten variable test problem n s — 132 and 
n e — 10000. The 132 sample sites were obtained from 
a .D-optimal experimental design used in previous 
work by Giunta, et al [2] and were located within the 
ten dimensional design space defined by [—1 l] 10 The 


variable test problem were the same as those used 
in the five variable test problem. For the Case 1 
test problem the DACE model had a correlation pa- 
rameter of 6 = 0.50 and /? = 2.6455, while for the 
Case 2 test problem these values were 0 = 0.50 and 
/ 3 = 4.7987. As before, the quadratic response surface 
polynomial models for both the Case 1 and Case 2 
test functions were created using the Fit [•] function 
in Mathematica. For the Case 1 RS/DACE model 
the optimal correlation parameter was 6 = 0.10, and 
/? = 4.69 • 10~ 15 , and for Case 2, these parame- 
ters were 0 = 0.10, and j3 = 7.31 • 10~ 15 . For the 
fourth model, Case 1 was y = 2.6214 and Case 2 was 
y = 4.7190. 

The results for the Case 1 and Case 2 test prob- 
lems are listed in Table 3. In Case 1, the polynomial 
RS model and the RS/DACE model exhibit nearly 
identical modeling errors and provide the best ap- 
proximations to the test function. The modeling er- 
ror for the DACE model is somewhat worse than for 
the polynomial-based models, and the modeling error 
for the mean- value model is the largest. 

While the results for the Case 1 test problem are 
similar for the five and ten variable versions of the test 
problem, this is not true for the Case 2 test problem. 


the DACE model. Note that once again the DACE 
model is only slightly more accurate than the mean- 
value model. 


4.4 Summary of Test Problem Results 

Note that some caution must be exercised in inter- 
preting these results as the modeling accuracy data 
and observations are applicable only to the Case 1 
and Case 2 test functions considered here. As may 
be expected, if different test functions had been inves- 
tigated, the results may have been different. In fact 
it is quite easy to create a test function for which 
the mean-value model is the most accurate model- 
ing method, as the authors discovered in some initial 
DACE modeling work. 

The results from the one variable test problem 
showed the expected trends, i.e., where the DACE 
model was more accurate for the Case 1 test func- 
tion and the polynomial RS model was more accu- 
rate for the Case 2 test function. However, the five 
and ten variable versions of the Case 1 test problem 
did t i Id th t d It. F th i 


the DACE model were only marginally more accurate 
than the mean-value model. Thus, the sinusoidal fea- 
tures of the test problem posed difficulties for both 
the polynomial RS and DACE models. 

For the one, five, and ten variable versions of the 
Case 2 test problem, it is clear that the polynomial 
RS model provides the highest modeling accuracy of 
the approximation methods considered in this study. 
These results were expected since the test function is 
quasi-quadratic. However, the most startling results 
are shown in the modeling error data for the DACE 
model as compared to the mean- value model for the 
ten variable test problem. Here, the DACE model 
is only slightly more accurate than the mean-value 
model . 


5 Conclusions 

In this study, the accuracy of quadratic polynomial 
models and DACE interpolating models was evalu- 
ated through the examination of several test prob- 
lems. The data obtained in this study showed that 
the quadratic polynomial models were more accurate, 


sory investigation is not intended to serve as an ex- 
haustive comparison between the two modeling meth- 
ods. 
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6 Future Work 

Clearly, there are numerous opportunities for further 
investigation, in both the formulation of the DACE 
models and in the examination of other test problems. 
For the test cases described in this study, future ar- 
eas of investigation include (1) the use of a vector of 
correlation parameters in the exponential correlation 
model, and (2) the examination of various methods 
to select sample sites in the design space. Both of 
these may significantly affect the modeling accuracy 
of DACE approximation models. 
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Table 1: Modeling errors in Cases 1 and 2 for the one variable test problem. 


Mean Median Std. RMS u b Max. 
Model Error Error Dev. Error Error 

Case 1 (e = 1.0) 

y(x) = p + Z(x) 0.051 0.027 0.045 0.068 0.203 

y(x)=c T x 0.076 0.057 0.076 0.107 0.328 

y(x) = y 0.081 0.063 0.058 0.100 0.184 

Case 2 (e = 0.7) 

y(x) = P + Z(x) 0.115 0.093 0.112 0.160 0.502 

y(x) = c T x 0.080 0.065 0.081 0.114 0.393 

y(x) = y 0.130 0.117 0.110 0.170 0.519 


Table 2: Modeling errors in Cases 1 and 2 for the five variable test problem. 



Mean 

Median 

Std. 

RMS u b 

Max. 

Model 

Error 

Error 

Dev. 

Error 

Error 

Case 1 (e = 1.0) 

y(x) = /?+ Z(x) 

0.211 

0.180 

0.159 

0.264 

0.992 

y(x) = c T x 

0.202 

0.171 

0.154 

0.254 

0.963 

y(x) = /(x) + p res , + Z(ves.) 

0.203 

0.181 

0.153 

0.254 

0.963 

j/(x) = y 

0.241 

0.210 

0.175 

0.298 

0.989 

Case 2 (e = 0.7) 

y(x) = p + Z(x) 

0.225 

0.190 

0.171 

0.282 

0.945 

y(x) = c T x 

0.210 

0.178 

0.158 

0.263 

0.944 

y(x) = f(x) + p res , + Z(res.) 

0.211 

0.179 

0.158 

0.264 

0.944 

2 /(x) = y 

0.696 

0.651 

0.425 

0.815 

1.793 


Table 3: Modeling errors in Cases 1 and 2 for the ten variable test problem. 



Model 

Mean 

Error 

Median 

Error 

Std. 

Dev. 

RMS u b 

Error 

Max. 

Error 

Case 1 (e = 1.0) 


y(x) = p + Z (x) 

0.651 

0.636 

0.362 

0.745 

2.010 


y(x) = c T x 

0.524 

0.477 

0.355 

0.633 

1.964 

2/(x) 

= f(x) + p res , + Z(res.) 

0.524 

0.479 

0.348 

0.629 

1.823 


II 

0.698 

0.696 

0.283 

0.753 

1.801 

Case 2 (e = 0.7) 


y(x) = P + Z(x) 

2.090 

1.920 

0.531 

2.157 

4.071 


y(x) = c T x 

0.380 

0.326 

0.218 

0.473 

1.646 

2/(x) 

= f(x) + pres. + Z(res.) 

0.544 

0.475 

0.416 

0.693 

2.566 


y(x) = y 

2.344 

2.385 

0.528 

2.403 

3.914 


11 

American Institute of Aeronautics and Astronautics 













Figure 3: A one dimensional view ol the Case 2 test 
Figure 1: A one dimensional view of the Case 1 test f unc t j on — q 7 ) 

function (e = 1.0). 


f(xl,x2) 



x2 ,0 


Figure 2: A two dimensional view of the smooth (top) . ... 

, • , p,, r , , , , r • , Figure 4: A two dimensional view of the smooth (top) 

and noisy variants ot the Case 1 test function (e = 0 . \ ' 

1 gx “ and noisy variants of the Case 2 test function (e = 

0.7). 
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f(xl) f(xl) 



xl 

Figure 5: The DACE and quadratic polynomial RS 
models for Case 1 (e 10) of the one variable test 



xl 

Figure 6: The DACE and quadratic polynomial R.S 
models for Case 2 (e = 0.7) of the one variable test 
function. 
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